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Abstract Body 


Background / Context: 

Description of prior research and its intellectual context. 

Developmental or remedial eollege courses are designed to help underprepared students. 
Although most of the literature examines developmental courses in community college, around 
18% of students at four-year institutions take developmental courses (Sparks & Malkus, 2013). 
We evaluate a developmental course in such a public institution. Using administrative transcript 
data from an elite state university and a regression discontinuity strategy, we estimate causal 
impacts of taking a pre-calculus course on future course decisions, major declarations, and 
college persistence. Our work pushes current literature in a new direction - using instructor 
characteristics to explore driving factors of these impacts. Preliminary results show that the 
recommendations to take pre-calculus have significant bite, and taking pre-calculus helps low 
performing students' grades in future calculus courses. Looking at course-specific outcomes 
elucidates students’ course-taking decisions and informs course-level academic policies. 


Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

Our paper asks whether placement recommendations for a developmental math course at an elite 
public institution impact students’ future academic performance, course-taking, and college 
outcomes. We use these specific outcomes to measure whether developmental courses help 
students develop the skills necessary to succeed in college, inspire them to take different courses, 
and overall help them graduate or persistent in college. 

In addition to examining student outcomes, we dig deeper than the current literature and examine 
the ways in which instructor characteristics can drive these outcomes. We ask whether 
instruction at this university in a program for low-achieving students and particularly 
underprepared low-income, first-generation, and underrepresented minority students achieves its 
goal of reducing achievement gaps. This informs specific course and instructor policies to help 
underprepared students in their first semesters in college. 

Setting: 

Description of the research location. 

Our research setting is an elite public institution. This institution is highly selective and has a 
large number of students, with an undergraduate enrollment of over twenty thousand. This 
institution house separate liberal arts, engineering, and business colleges. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 

We collected? information on all students and instructors who enrolled or taught in all courses at 
this institution from 2002 to 2015. 

Significance / Novelty of study: 

Description of what is missing in previous work and the contribution the study makes. 

Our study advances the literature studying developmental courses in three ways. First, we study 
the context of an elite four-year public institution, whereas the previous literature has focused on 
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community colleges (Bettinger & Long, 2005,2009; Boatman & Long, 2010; Clayton & 
Rodriguez, 2012). At an elite institution with higher quality resourees, it is extremely important 
for underprepared students to master the skills neeessary to take advantage of the higher equality 
edueation. 

Seeond, we use student transeript data to traek student eourse-taking as a result of these 
developmental eourses. Previous literature largely foeuses on eollege persistenee, dropout, and 
aeeumulated eredits. An additional measure to augment these is eourse-taking; The literature has 
not answered whether students who take developmental eourses are more likely to take Seienee 
Teehnology Engineer Mathematies (STEM) eourses, or whether dropping out of eollege is 
related to the eourses students eoneurrently taken with developmental eourses. Answering these 
questions better identifies how developmental eourses impaet students’ aeademie deeisions, and 
ean eneourage STEM partieipation. 

Third, we leverage instruetor data to explore faetors for measuring impaets, informing poliey to 
help students early in eollege. Instruetors ean have signifieant effeets on student outeomes sueh 
as major deeisions (Ost & Main, 2014) and persistenee (Priee, 2010). Previous literature has 
found instruetor impaets in general (Hoffman & Oreopoulus, 2009; Eiglio, Sehapiro, & Soter, 
2015), but our work is the first to look at instruetor impaets in developmental eourses. This is 
partieularly important for developmental eourses, whieh are intended to have lasting impaets. 

Our foeus on a program speoifieally targeting underprepared students seeks to determine the 
faetors that ean inerease the likelihood of sueeess for these students. 

Statistical, Measurement, or Econometric Model: 

Description of the proposed new methods or novel applications of existing methods. 

A major estimation hurdle is that students who take developmental eourses are different from 
students who do not. A simple eomparison of students who take developmental eourses against 
students who do not is not an “apple-to-apples” eomparison. Eor example, students with high 
SAT math seores most likely do not take developmental eourses and eannot be eompared against 
students who do. We use a regression-diseontinuity (RD) framework to overeome this hurdle and 
use a fair eomparison of students. 

With our RD strategy, we ean measure the eausal impaet of reeeiving a reeommendation for 
developmental eourses. In our eontext, students are reeommended to take a math developmental 
eourse based on explieit eutoffs in a ealeulated index. The RD framework argues students in a 
narrow interval around the eutoffs are indistinguishable. This allows an “apples-to-apples” 
eomparison of students who just managed to reeeive a reeommendation to students who did not. 
Using this RD framework, we ean identify the eausal impaet of reeeiving the reeommendation to 
take a math developmental eourse. 

Usefulness / Applicability of Method: 

Demonstration of the usefulness of the proposed methods using hypothetical or real data. 

One key assumption in RD is that students are indistinguishable around the eutoff. We test this 
assumption by plotting the eharaeteristies of the students as a funetion of the ealeulated index 
and the eutoff (MeCall & Biebly, 2012; Imbens & Eemieux, 2007). This index is ealeulated 
based on students’ pre-eollege test seores: high sehool grade point average, SAT or ACT math 
seores, and a math plaeement exam. Therefore we expeet to see a smooth eorrelation between the 
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index and student eharaeteristies, but no diserete ehanges around the eutoff. We show several 
plots in Figures (1) - (3). These figures plot how the pereent of blaek students, ACT and SAT 
math seores vary with the ealeulated index. It also plots predieted linear regressions of the index 
on these eharaeteristies. These figures show the proportion of being blaek, ACT math and SAT 
math seores do not diseretely ehange at the eutoff. The figures for other student eharaeteristies 
look similar, showing a smooth pattern through the eutoff. 

Another key assumption in RD is students eannot easily manipulate their indexes and thus the 
ealeulated reeommendations. We do not believe this a large issue in this study. 

Reeommendations are ealeulated through the Offiee of the Registrar, using a formula unknown 
to students. In addition, these reeommendations are not binding and students are free to ignore 
them, eliminating any ineentive to manipulate the reeommendations. 

Research Design: 

Description of the research design. 

The first step to applying our RD to the data is to reereate students’ reeommendations. We 
gathered the administrative formulas the Offiee of the Registrar used to ealeulate student 
reeommendations, as well as aeeessed the student administrative data. We show in Figure (4) we 
are able to reproduee student reeommendations. This verifies we are using the eorreet student 
data and applying the eorreet administrative formulas to ealeulate the index. 

Next we seleet our sample to eontain only students with ealeulated indexes in a narrow interval 
around the eutoffs used to determine reeommendation. There are four mutually exelusive 
ree ommendations : 

1. Definitely Take Pre-Caleulus; 

2. Tentative Take Pre-Caleulus; 

3. Tentative Take Caleulus I; and 

4. Definitely Take Caleulus I or Higher. 

Not only are there reeommendations for different eourses, but reeommendations eome in varying 
strengths. For our analysis, we foeus on the differenee between Definitely and Tentatively Take 
Pre-Caleulus. In future analysis, we will foeus on the differenee between Tentatively Take Pre- 
Caleulus and Tentative Take Caleulus I. 

Finally, we use a linear regression on our seleeted sample to estimate the eausal impaet of the 
reeommendation. We inelude the index distanee above and below the eutoff as additional 
eontrols. For a student f we estimate the eausal impaet of the reeommendations on a host of 
outeomes Tj: 

= 

a{Receive Recommendatiori)i -I- (Index Distance Below Cutoff)i -I- 
P20'iT-dex Distance Above Cutoff)i 

Where a is our eoeffieient of interest, a measures the differenee in outeome Tj between students 
who just managed to reeeive the reeommendation and students who did not. In an RD 
framework, these students are indistinguishable. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 
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We gathered administrative student and instruetor data from the Offiee of the Registrar. These 
data eontain reeords for eaeh student and instruetor who partieipated in elass taught at this elite 
publie institution from 2002 to 2015. Student data ineludes not only standard demographie and 
admission test data, but also detailed transeript data. This allows us to traek students’ entire 
eourse history while at this institution. Transfer eredit reeords are also eolleeted. This is an 
ineredibly rieh dataset that allows us to measure a large variety of student outeomes, sueh as how 
many Aeeounting, Biology, Eeonomies, or Psyehology eourses the student has taken in any 
given semester. 

The instruetor data is equally as rieh. It eontains demographie eharaeteristies of instruetors, as 
well as their status in the institution (faeulty, leeturer, graduate student, ete.). For graduate 
student instruetors, GRE and TOEFE seores are available. We are partieularly interested in 
instruetor evaluation data. We use instruetors’ evaluation histories to see whether instruetor 
teaehing quality ratings are related to student outeomes through the developmental eourses. We 
are also able to relate the quality ratings in the developmental eourse to performanee in 
subsequent elasses. 

Findings / Results: 

Description of the main findings with specific details. 

Our preliminary findings ean be found in Figure (4). Elsing the linear regression and RD 
framework, we find students who reeeive a Definite Reeommendation to take Pre-Caleulus are 
more likely to take Pre-Caleulus eompared to students who reeeive only a Tentative 
Reeommendation. This shows these reeommendations have a signifieant effeet on students’ 
eourse-taking deeisions, and is our first step to looking at how these reeommendations affeet 
how students ehoose other eourses. 

In future work, we will measure effeets on other eourses, sueh as whether students who reeeive a 
Definite Reeommendation are less likely to take STEM eourses. This allows us to see students’ 
eourse-taking trajeetories when they reeeive different reeommendations. It also elueidates how 
students may ehoose to eventually drop out or persist in eollege. 

To explain possible faetors for these impaets, we will look for instruetor effeets among those 
who teaeh the Pre-Caleulus and Caleulus I eourses as well as the effeets of the university’s 
program for underprepared students. The quality of instruetion may eontribute to the impaet on 
students, and the university assumes that the program has benefieial effeets. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

To our knowledge, our work is the first to evaluate developmental eourses at an elite publie 
institution. Our findings will inform polieies to help underprepared students at higher quality 
institutions, where the benefits of mastering the eollege skills are higher than in eommunity 
eolleges. Our results on student eourse-taking links how students start eollege to eollege 
persistenee and dropout. These developmental eourses ean be used not only to help students 
through eollege, but also enable them to enter speeifie fields they might otherwise avoid. Finally, 
our results on individual instruetor and developmental program effeets will help to inform 
training and hiring deeisions to further inerease the ehanees underprepared students will sueeeed. 
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ACT math 


Appendix B. Tables and Figures 

Figure 1: Average ACT Math Score over the Calculated Index 



Calculated Index 


Predicted ACT_math within Cutoffs • Average over Bins of .02 


Vertical lines at 1.5, 2, and 2.25 are the cutoffs for Placement Recommendations 
Bins of .02 were used to create average of ACT_math, strictly for scatter and not line plot. 
Calculated Indices below zero are not shown for graphical purposes 
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Figure 2: Proportion of Black Students over the Calculated Index 



Calculated Index 


Predicted Black within Cutoffs ♦ Average over Bins of .02 


Verticai iines at 1.5, 2, and 2.26 are the cutoffs for Piacement Recommendations 
Bins of .02 were used to create average of Black, strictly for scatter and not line plot 
Calculated Indices below zero are not shown for graphical purposes 


SREE Spring 2016 Conference Abstract — Hsu and Gehring 


B-2 


SAT_math 

400 500 600 700 800 


Figure 3: Average SAT Math Score over the Calculated Index 



Calculated Index 


Predicted SAT_math within Cutoffs ♦ Average over Bins of .02 


Vertical lines at 1.5, 2, and 2.25 are the cutoffs for Placement Recommendations 
Bins of .02 were used to create average of SAT_math, strictly for scatter and not line plot 
Calculated Indices below zero are not shown for graphical purposes 
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Proportion of Students Ever Taking Pre-Calculus 


Figure 4: Preliminary Results of Ever Taking Pre-Calculus 


Comparing Students with Definite (Left) and Tentative (Right) 
Recommendations to take Pre-Calculus 



Predicted Pre-Calculus Attendance within Cutoffs 


Vertical line at 1.5 is the cutoff for Placement Recommendation 
Includes scatter plot within intervals of .02 
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